Patent rating

ABSTRACT

Various embodiments of the present disclosure include systems and methods of assigning internal and external ratings to a subject patent owned by a subject entity. In one example embodiment, the method comprises assigning an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and assigning an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.

RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. 119(e) of U.S. Provisional Patent Application Ser. No. 61/542,545 filed Oct. 3, 2011, which is incorporated herein by reference in its entirety and made a part hereof. This application also claims the benefit under 35 U.S.C. 119(e) of U.S. Provisional Patent Application Ser. No. 61/561,502 filed Nov. 18, 2011, which is incorporated herein by reference in its entirety and made a part hereof.

BACKGROUND

The management of a patent portfolio involves multiple stages. Initially, a decision is typically made as to what inventions are worth the investment of filing a patent application. Then, each filed patent application goes through prosecution with the patent office. Finally, for each patent that is allowed, maintenance fees are usually payable at a variety of intervals to keep the patent in force.

SUMMARY

This disclosure relates to the creation of internal and external ratings for a patent.

In one example embodiment, an internal rating for a subject patent is created or assigned based on the relevance of the subject patent to the key technologies and/or patents of a subject entity or company, typically the patent owner or an assignee of the patent. A second or external rating is created or assigned based on the patent's relevance to key technologies and/or patents of one or more third parties external to the subject entity, typically an external competitive company.

An internal patent rating for the subject company, such as the patent owner for example, may be created by identifying automatically or manually the best of those patents owned by that company. Patents owned by the subject company are referred to herein as subject patents. The best subject patents may be selected based on factors such as known company product coverage, patent marking lists, historical patent litigation outcomes, or by looking at how many times the patent has been cited against other patents, for example. These patents may be ascribed a top rating, or score, such as a number in the range 8-10, for example.

Further patents of the subject company may be rated and ascribed a relative rating by their relevance to the best company patents identified above. Other parameters may also be used. The relative ratings may be any number in the range 0-7, for example.

An external rating for a patent may be created based on the patent's relevance to key technologies and patents of one or more third parties external to the subject company, typically an external competitive company. The creation of an external rating may also or in the alternative, include identifying the one or more of the best patents owned by that external company, and ascribing a relative rating of any given patent of the subject company based on its relevance to or relative worth when compared against the best patents of the competitor company.

For external ratings, the relevance of any given patent of the subject company (for example, a subject patent on which an annuity is due) to each competitor is rated. In this way, a more complete picture of a patent's worth is provided. A patent that may have a very low internal rating and hence at first blush appear to be a good candidate for abandonment may nevertheless have a high external rating indicating that payment of further maintenance fees is warranted.

The external third parties may be manually identified, or further determined, for example, by which parties have patents in the same class or subclass as the subject patent or patent owner's portfolio of patents, or identified by persons who are cited as inventors or companies identified as assignees of patents cited against the subject patent.

The ratings may be used in a patent and annuity management systems in which annual goals are established for each year or period. Goals such as the number of new patent applications, the number of patents to issue, the former and latter determined by country, the number of invention disclosures, the number of defensive publications, and so forth may be presented to a user in a dashboard or provided in a user interface or user terminal, for example.

Example EMBODIMENTS

In accordance with an example embodiment of this disclosure, there is provided a system for rating a subject patent owned by a subject entity, the system comprising one or more processors to receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.

The one or more processors may be configured to use one or more internal factors selected from the group comprising perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

The one or more processors may be configured to use one or more external factors selected from the group comprising perceived deterrent value of the subject patent against the one or more third parties; third party product coverage; historical patent litigation outcomes involving the one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

In accordance with another example embodiment of this disclosure, there is provided a system for rating a subject patent owned by a subject entity, the system comprising a network; at least one database, accessible on the network, the database including at least one database of patents owned by the subject entity; a server, operatively connected to the network, wherein the server includes: a processor, a memory, software operable on the processor to receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on selected external factors relating to one or more third parties external to the subject entity.

The software operable on the processors may use one or more of the internal or external factors listed above in automatically assigning the internal and external ratings for the subject patent.

BRIEF DESCRIPTION OF DRAWINGS

Some embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which:

FIG. 1 is a schematic view of computer network system 100, according to various embodiments.

FIG. 2 is a user interface, as may be used in an example embodiment.

FIG. 3 is a block diagram of a system, according to example embodiments.

FIG. 4 is a flow chart showing a method, according to an example method embodiment.

FIG. 5 is a block diagram of a machine in the example form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies herein discussed.

DETAILED DESCRIPTION

The life cycle of a patent may include multiple stages. These stages generally include invention, filing a patent application on the invention, prosecuting the patenting application to allowance or abandonment, determining whether to file any continuing applications, and paying maintenance fees on the allowed patent.

At each stage, one or more parties determine the best course of action to take with respect to the invention. For example, when determining whether or not to file a patent, the inventor may know what products are out in the technology area of the invention, a business manager may know how the invention fits in with a company's goals, and a patent attorney may have researched existing patents or applications in the technology area of the patent. These parties will ultimately decide whether it is worth the initial investment of filing an application on the invention and then determine the next course of action at each future stage.

In some instances, an inventor, business manager, or portfolio manager may wish to monitor the patent activities of one or more competitors. The provision of up-to-date information relating to cited references or technology trends in the art to which a user's or competitor's patent portfolio relates may be particularly helpful in devising patenting strategy or making informed business decisions.

In various embodiments, a patent management system includes tools to help the parties involved in the patenting process make decisions at each stage in the life of a patent. These tools may also be used for general research by parties not immediately involved with the patenting of the invention. These tools may be used as standalone tools, in combination with other tools, and in combination with other patent management systems. Examples of tools include, but are not limited to, prosecution analytics, patent and patent continuation analysis, reference management, prior art analytics, docketing management, claim mapping, claim analytics, portfolio analytics, external database (e.g., Patent Application Information Retrieval “PAIR”) analytics, annuity management, strategic monitoring, and a white space indicator.

The systems, methods, and tools of the present disclosure set forth in this specification are described in relation to a patent management system and patent matters, but it will be understood that embodiments of the present invention could equally be applied to other forms of intellectual property (trademarks, copyright, utility models, petty patents, registered designs, and the like). Moreover, the term “patent” is not intended to be limited to an issued patent, but may include a pending patent application or un-filed application or invention disclosure. The term “user” or “registered user” is intended to cover any person interacting with the patent management system. A user may be an inventor, portfolio manager, business manager or patent attorney, for example. A “user” or “registered user” may also be an annuity payment service provider such as Computer Packages, Inc (CPI) or Computer Patent Annuities (CPA), for example. In the patent field the term “annuity fees” is sometimes used to refer to payments required to renew granted patents, while the term “maintenance fees” is sometimes used to refer to payments required to maintain pending patent applications. In this specification, the terms “annuity fees” and “maintenance fees” are used interchangeably and refer to any kind of payment required to keep a patent or patent application (or other intellectual property right) in force.

FIG. 1 is a schematic view of computer network system 100 according to various embodiments. The computer network system 100 includes a patent portfolio and annuity management system 102 (also called a portfolio management system 102, or annuity management system 102) and user terminal 104, communicatively coupled via network 106. In an embodiment, patent portfolio and annuity management system 102 includes web server 108, application server 110, database management server 114, which is used to manage at least operations database 116, and file server 118. Patent portfolio and annuity management system 102 may be implemented as a distributed system; for example, one or more elements of the patent portfolio and annuity management system 102 may be located across a wide area network (WAN) from other elements of patent portfolio and annuity management system 102. As another example, a server (e.g., web server 108, file server 118, database management server 114) may represent a group of two or more servers, cooperating with each other, provided by way of a pooled, distributed, or redundant computing model.

Network 106 may include local area networks (LANs), WANs, wireless networks (e.g., 802.11 or cellular network), the Public Switched Telephone Network (PSTNs) network, ad hoc networks, personal area networks (e.g., Bluetooth) or other combinations or permutations of network protocols and network types. The network 106 may include a single LAN or WAN, or combinations of LANs or WANs, such as the Internet. The various devices coupled to network 106 may be coupled to network 106 via one or more wired or wireless connections.

Web server 108 may communicate with file server 118 to publish or serve files stored on file server 118. Web server 108 may also communicate or interface with the application server 110 to enable web-based presentation of information. For example, application server 110 may provide scripts, applications, or library files that provide primary or auxiliary functionality to web server 108 (e.g., multimedia, file transfer, or dynamic interface functions). In addition, application server 110 may also provide some or the entire interface for web server 108 to communicate with one or more of the other servers in the patent portfolio and annuity management system 102 (e.g., database management server 114). Web server 108, either alone or in conjunction with one or more other computers in patent portfolio and annuity management system 102, may provide a user-interface. The user-interface may be implemented using a variety of programming languages or programming methods, such as HTML (HyperText Markup Language), VBScript (Visual Basic® Scripting Edition), JavaScript™, XML® (Extensible Markup Language), XSLT™ (Extensible Stylesheet Language Transformations), AJAX (Asynchronous JavaScript and XML), Java™, JFC (Java™ Foundation Classes), and Swing (an Application Programming Interface for Java™).

User terminal 104 may be a personal computer (PC) or mobile device. In an embodiment, user terminal 104 includes a client program to interface with the patent portfolio and annuity management system 102. The client program may include commercial software, custom software, open source software, freeware, shareware, or other types of software packages. In an embodiment, the client program includes a thin client designed to provide query and data manipulation tools for a user of user terminal 104. The client program may interact with a server program hosted by, for example, application server 110. Additionally, the client program may interface with database management server 114.

Operations database 116 may be composed of one or more logical or physical databases. For example, operations database 116 may be viewed as a system of databases, which when viewed as a compilation, represent an “operations database.” Sub-databases in such a configuration may include a matter database, portfolio database, user database, patent reference document or prior art database, product database, watch database, and mapping database. The operations database 116 or any one of the sub-databases may include a registry of patent matters. Operations database 116 may be implemented as a relational database, a centralized database, a distributed database, an object oriented database, or a flat database in various embodiments.

Data stored in a first database may be associated with data in a second database through the use of common data fields. For example, consider entries in the patent database formatted as [Patent ID, Patent Number] and entries in the product database formatted as [Product ID, Product Name, Patent ID]. In this manner, a product entry in the product database is associated with a patent in the patent database through the Patent ID data field. In various embodiments, a product may be associated with more than one patent by creating multiple entries in the product database (one for each patent the product is associated with). In other embodiments, one or more patent reference document may be associated with a patent by creating multiple entries in the patent database, for example. The formats and data field titles are for illustration purposes, and other names and formats may be used. Additionally, further associations between data stored in the databases may be created as discussed further herein.

During operation of patent portfolio and annuity management system 102, data from multiple data sources (internal and external) is imported into or accessed by the operations database 116. Internal sources may include data from portfolio management and annuity payment services affiliated with patent portfolio and annuity management system 102. External sources 120 may include websites or databases associated with foreign and domestic patent offices (e.g. PAIR at the United States Patent and Trademark Office “USPTO”), assignment databases, World Intellectual Property Organization (“WIPO”), and International Patent Documentation Center (“INPADOC”). In an embodiment, annuity amounts due and annuity due dates for patent matters listed in a registry database are drawn directly from such data stored at a national patent office in electronic form. In various embodiments, the data is scraped and parsed from websites if it is unavailable through a database. The data may be gathered using API calls to the sources when available. The data may be imported and stored in the operations database 116 on a scheduled basis, such as weekly, monthly, quarterly, or some other regular or periodic interval. Alternatively, the data may be imported on-demand.

After data importation, the data may be standardized into a common format. For example, database records from internal or external sources 120 may not be in a compatible format with the operations database 116. Data conditioning may include data rearrangement, normalization, filtering (e.g., removing duplicates), sorting, binning, or other operations to transform the data into a common format (e.g., using similar date formats and name formats).

FIG. 2 is an example user interface 200 of patent portfolio and annuity management system 102 (FIG. 1). User interface 200 is illustrated with multiple user interface elements. In an example embodiment, a user interface element is a graphical or textual element with which a user may interact to cause an application to perform an assigned action for the interface element. Data representing user interface 200 may be transmitted via network 106 and presented on a display of user terminal 104 (FIG. 1) through the use of a web browser. A user (e.g., manager of a patent portfolio) may interact with the user interface elements of user interface 200 through the use of an input device (e.g., stylus, cursor, mouse, finger) of the user terminal 104. In an embodiment, a user selection is based on the coordinates of the input device as it makes contact with the display or where a user “clicks” the mouse. The coordinates are compared to the coordinates of the user input element to determine the selection. The type of user elements, names, and layout depicted in FIG. 2 are intended to be an illustration of an example user interface of patent portfolio and annuity management system 102. Other types of user elements, names, and layouts may be used.

The user interface elements may include my matters 202, patent watches 204, company watches 206, patent analysis 207, and tag matters 209. These elements may be used to select a context/view of the portfolio and annuity management system 102 (FIG. 1). For example, my matters 202 may list the patents included within a portfolio of a user, or list patent matters tagged for renewal by a user of an annuity registry database included, for example, in an operations database 116 (FIG. 1) of the type described above. Patent watches 204 may list information on patents that the user has indicated the management system 102 is to watch, and company watches 206 lists information on companies that the user has indicated the system should watch. Patent analysis 207 may allow a user to perform analysis on selected patent or patent continuation matters. Tag matters 209 may be selected to allow a user to tag patent matters in a patent registry for payment of annuity fees. Further details of each of these are included herein.

Date boxes 208 and 210 are user elements that allow a user to select a time period, for example to select a display of patents having annuity fees due in that time period. Amount due box 212 displays the amount due with respect to annuities for patents in the portfolio of the user, or tagged or listed in a registry, within the period indicated by date boxes 208 and 210. Upon activating (e.g., clicking) one of the date boxes, a user may be presented with a calendar, which allows the selection of a date. Upon selecting a date, the date boxes will update to reflect the user's choice.

Drop-down menu 214 includes a list of portfolios or registry listings that a user of the patent portfolio and annuity management system 102 (FIG. 1) is authorized to view. For example, before user interface 200 is displayed, a login screen requesting a user ID and password may be presented to the user. In various embodiments, the user ID is associated with one or more portfolios. In turn, each portfolio is associated with one or more matters. Matters may include US and foreign issued patents (including divisionals, continuations and so forth), pending patents, abandoned patents, and not yet filed applications. Thus, upon selection of a portfolio using drop-down menu 214, user interface 200 is populated with matters associated with the portfolio or registry listing.

Checkbox 216 is an option to only display matters that currently have an annuity due. Drop-down menu 218 allows further filtering of matters. For example, the matters may be filtered by US patents only, by US patent applications, and by foreign applications only.

In various embodiments, activation of button 220 updates user interface 200 to reflect the choices made by the user with respect to date boxes 208 and 210, checkbox 216, and drop-down menu 218. For example, the amount due box 212 will be updated to reflect the amount due within the new period, and the matters listed under column headings 222 may be filtered. In various embodiments, user interface 200 is updated as the user selections are made with respect to elements 208 to 218 without activating button 220.

FIG. 3 is a block diagram of patent portfolio and annuity management system 102 (FIG. 1), according to an example embodiment. Illustrated are user database 302, portfolio database 304, matter database 306, watch database 308, product database 310, analytics database 312, input module 314, parsing module 316, analytics module 318, display module 320, filtering module 322, mapping module 324, and payment module 326. In various embodiments, the data stored in databases 302, 304, 306, 308, 310, and 312 may be in the same or multiple physical locations. For example, portfolio database 304 may be located in one or more computers associated with a portfolio management service. In various embodiments, patent portfolio and annuity management system 102 (FIG. 1) may minor databases stored in other locations. In an embodiment, when a request is made to access data stored in the databases, patent portfolio and annuity management system 102 determines where the data is located and directs the request to the appropriate location.

In an example embodiment, user database 302 stores data representing users of patent portfolio and annuity management system 102 (FIG. 1). The data may represent authorized or registered users that are registered to use the patent matter registry database, for example. The data may include data fields of user credentials, including a user ID and password, and access rights with respect to the management system 102. The user ID may be the same as the user ID of other systems associated with patent portfolio and annuity management system 102. For example the user ID may be used in a portfolio management system 102. In this manner, access rights of the user with respect to the portfolio management system 102 may follow to the patent portfolio and annuity management system 102. In various embodiments, each user ID is associated with one or more portfolio IDs. Thus, when a user logs into the patent portfolio and annuity management system 102, the user is presented with access to only the portfolios that have been associated with that user ID. More or fewer data fields associated with a user may be included in a user entry stored in user database 302.

Portfolio database 304, in an example embodiment, stores data representing portfolios of one or more matters. A matter may include foreign or domestic issued patents in addition to applications. Data stored in portfolio database 304 may have been initially generated in a portfolio management system and imported into patent portfolio and annuity management system 102 through the use of one or more API calls or direct access to the data in the portfolio management system. In various embodiments, a portfolio may be generated by a user using the patent portfolio and annuity management system 102. For example, a user interface may be presented to the user requesting a portfolio name and the identifiers of matters to be included in the portfolio. In an embodiment, a portfolio entry in portfolio database 304 includes the data fields of portfolio ID and portfolio name. Additionally, a data field for matter ID may also be included in an entry in the portfolio database. Thus, each portfolio may be associated with one or more matters through the use of the matter ID data field. More or fewer data fields associated with a portfolio may be included in a portfolio entry of portfolio database 304.

In an embodiment, matter database 306 stores data representing matters. Each matter may be associated with one or more portfolios as well as one or more watches. In some embodiments, a matter is associated with no portfolios or watches. In an embodiment, a matter entry includes data fields representing a matter ID, title (e.g., the name of the patent or application), type of the matter (e.g., application, issued patent, Patent Cooperation Treaty “PCT” application), status of the matter (e.g., issued, abandoned, allowed), a link to the patent office where the matter was filed, a link to a PDF download of the matter, abstract of the matter, inventors of the matter, current owner of the matter, cited references on the face of the matter, filed date, issue date, docket number, and annuity information (e.g., due date, country, and amount due). In some embodiments, other patent reference documents or prior art in any form may be stored and associated with one or more matters. More or fewer data fields associated with a patent may be included in a matter entry stored in matter database 306. In an example embodiment, matter database 306 may store a patent matter database, wherein this database includes data about the patent matters. The data may include, for at least one patent matter, a claim set or statement of invention and a priority date for the claim set or statement of invention. Matter database 306 may also store a database of prior art documents (also known as “references”), wherein the prior art database includes data about the prior art documents. The data may include, for at least one prior art document, a priority date or publication date of the document. One or more of the prior art documents may be associated with a first patent matter in the patent matter database.

In various embodiments, a matter is associated with one or more other matters as a family with a family ID. Family members may be priority documents, continuation patents/applications, divisional patents/applications, and foreign patent/application counterparts. In an embodiment, family information is determined according to an external source such as INPADOC. Patent reference documents and/or other prior art may be manually or automatically stored, cross-cited, and associated with related family matters, for example.

In an embodiment, watch database 308 stores data representing user preferences with respect to objects that the user wants to track (e.g., watch). In an embodiment, an object is a company or a matter. With respect to matters, a user may indicate a preference to track the matter for any forward citations within a time period (e.g., 60 days, 3 months, etc.). Thus, watch database 308 may store entries with a watch ID that is associated with a user ID and a matter ID. With respect to companies, a user may indicate a preference to watch for publications or issued patents with an associated time period. Thus, watch database 308 may store entries with a watch ID, a user ID, and a company name. In various embodiments, the watches are user ID specific such that no matter what profile the user is viewing, the watches do not change.

In an embodiment, product database 310 stores data representing products. The products may be actual products manufactured by the user, products yet to be released, planned products, products covered in a licensing agreement, or a fictional product. In various embodiments, product database 310 stores entries with a product ID and product name which are associated with a user ID. In various embodiments, a user may tag a product as associated with a matter. Thus, an entry in the product database 310 may also be associated with a matter ID. In an embodiment, one or more products stored in the product database 310 may be mapped to one or more matters stored in the matter database 306.

In various embodiments, analytics database 312 stores data representing information on matters and companies associated with patent portfolio and annuity management system 102 (FIG. 1). This information may be organized according to an individual matter, a portfolio, a family, or company. The information may be gathered from internal or external sources 120 of patent portfolio and annuity management system 102 or may be generated by examining the data stored in one or more databases of the management system 102.

In various embodiments, information for an individual matter may include status (e.g., disclosure received, drafting, filed, completed-waiting examination, in prosecution, allowed, issued), cited prior art, list and type of rejections (e.g., 35 U.S.C. §101, 102, 103), number of claims allowed, office action count, interviews held, Information Disclosure Statements (“IDS's”) filed, attorney fees to date, and PTO fees to date. Further metrics for an individual matter stored in analytics database 312 may include time waiting for examiner, total time in examination, time waiting for PTO while in examination, and time on appeal. An additional prosecution details formatter may include upcoming docket dates (e.g., due dates for office action, IDS due dates, etc.). A claim prosecution history chart may also be stored in analytics database 312 for each matter. The prosecution history may include a history of claim amendments and arguments made in prosecution.

In various embodiments, analytics for an individual matter may also include data representing a claim chart. The claim chart may include one or more scope concepts on one axis and claims on the other axis. In an embodiment, a scope concept is a textual description of what a patent claim is at least limited to. Thus, if a claim comprises A, B, and C, a scope concept may be A. In various embodiments, a scope concept for the claim may be broader than A because the claim will still be limited to the broader scope concept. In an embodiment, the claim charts store mappings between claims and scope concepts. The claim chart may include a variety of levels of granularity of scope concepts. Some claims may be mapped to all of the scope concepts while others may not be mapped to any scope concepts. In various embodiments, the mappings may have already been generated in a program external to the management system 102 (FIG. 1) (e.g., a portfolio mapping system). In various embodiments, claim charts may be combined at the portfolio level. Thus, a portfolio level chart may include scope concepts for all the claims (or, in some examples, only the independent claims) in the portfolio.

In various embodiments, portfolio and family analytics information may be stored in analytics database 312 based on aggregating the metrics for an individual matter. In an embodiment, the analytics of a family may differ from those in a portfolio since not all family member matters may be part of a portfolio. Additionally, the analytics may be stored for multiple time periods such that comparisons may be made between current metrics and metrics from a year ago (or other time period).

Aggregated metrics may include total patents and applications, pending and waiting for examination, in prosecution with no claims allowed, in prosecution with some claims allowed, appealed, notice of allowance received, and issued. These metrics may be compiled for both US and international matters in the portfolio. Additional information may include the international portfolio distribution by country. An age distribution of the portfolio may also be compiled. For example, the portfolio may be broken down by less than one year old, 1 to 3 years old, 3 to 5 years old, 5 to 10 years old, 10 to 20 years old, and 20 or more years old. Recent filing and issue activity as well as upcoming foreign filing deadlines may be stored in analytics database 312.

In various embodiments, prosecution metrics across a portfolio or family are determined and stored in analytics database 312. This data may include matters waiting for examination, matters in examination, matters appealed, and cases allowed but not yet issued. Analytics with respect to office actions may also be compiled and stored. For example, success of allowance after a first office action response, a second response, and the like, may be stored in analytics database 312. Additional prosecution analytics may include allowance rate (e.g., 10%) after a telephone interview, allowance after an in-person interview, percentage of interviews with response, number and type of rejections, average time in prosecution (e.g., in months), average time to file a response, and average time to next office action. Further, the list of recently allowed cases, recently appealed cases, stalled cases, and recently abandoned cases may also be stored for each portfolio and family in analytics database 312.

In various embodiments, data stored in the database for a group of matters in a portfolio or family is analyzed to determine the top (e.g., ten) most cited patents, top most cited inventors, top most cited prior art owners (e.g., according to assignment documents or the face of the patent), newly (e.g., within the last 60 days) cited prior art owners, and top most cited prior art inventors. This information may be stored in analytics database 312.

In various embodiments, annuity cost information regarding cost projections for both US issued patents and international patents may also be stored. Upcoming foreign filing deadlines with respect to annuities may also be stored in analytics database 312.

In various embodiments, the results of keyword analysis on one or more matters and/or prior art references may be stored in analytics database 312. The keyword analysis may be based on the occurrences of the keywords in the matter or references to derive a score or keyword overlap.

In various embodiments, input module 314 receives data from multiple sources where it may be further processed by one or more other modules and stored in one or more of databases 302-312. For example, input module 314 may be configured to utilize one or more APIs to collect data from one or more external sources 120 (FIG. 1) (e.g., public PAIR, private PAIR, INPADOC, foreign patent offices, patent docketing systems, portfolio management systems, etc.). The data may include published patent documents (for example, published claims of a patent), issued patent documents (for example, granted claims of a patent), patent applications, office actions or other patent office correspondence, prior art references, claim mappings, dockets dates, and annuity payment data.

In various embodiments, input module 314 is configured to receive input from one or more user interface elements. For example, the patent portfolio and annuity management system 102 (FIG. 1) may present multiple user interfaces to users. These user interfaces may enable users to input data directly into databases 302-312, instruct the management system 102 to retrieve data from patent data stores, and instruct the management system 102 to perform various operations on the data in databases 302-312.

Additionally, input module 314 may be configured to determine the selection of one or more user interface elements by a user and initiate the action associated with the selected user interface element. For example, a user interface element may include a drop-down menu to select a portfolio. Input module 314 may be configured to receive a selection of the portfolio and a drop-down menu. Then, input module 314 may pass the selection to one or more other modules for further processing. For example, display module 320 may update the drop-down menu to indicate the selection of a portfolio.

In various embodiments, parsing module 316 takes the data that has initially been inputted by input module 314 and formats it according to the data fields of databases 302-312 as discussed above. For example, consider a patent publication that has been inputted by input module 314. Parsing module 316 may use a combination of automatic image recognition and text analysis to determine the filing date, issue date, title, abstract, and claims of the patent. In some embodiments, parsing module 316 may flag certain pieces of data that had been determined to be potentially inaccurate (e.g., a number that could not be read). A user of patent portfolio and annuity management system 102 may then examine the flagged data and manually enter in the information.

The resulting data that has been parsed by parsing module 316 may then be entered as an entry in one or more of databases 302-312. This may be accomplished by, for example, formulating an SQL query with the parsed information. In various embodiments, parsing module 316 may parse multiple pieces of information before generating a database entry. For example, input module 314 may receive a docket number for an issued patent. The docket number may be combined with the information parsed from the issued patent to form an entry in matter database 306.

In various embodiments, analytics module 318 is configured to examine and run calculations on the data stored in the databases 302-312 to generate the analytics discussed previously. For example, analytics module 318 may formulate an SQL query that retrieves the number of times that a prior art reference has been cited within a portfolio. This query may be run for each prior art cited within the portfolio to determine a list of the most cited (e.g., the top ten) prior art references within a portfolio. In an embodiment, the queries are formulated and run as requested by a user. In an embodiment, once the analytics information has been determined, it is stored within analytics database 312. In various embodiments, queries are formulated and run on a period basis (e.g., nightly) and entries in analytics database 312 may be updated to reflect any changes.

In various embodiments, the analytics module 318 may be used for automatically rating a subject patent owned by a subject entity. One or more processors of the analytics module 318 may receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.

The one or more processors may use one or more internal factors selected from the group comprising: perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

The one or more processors may use one or more external factors selected from the group comprising: perceived deterrent value of the subject patent against one or more third parties; third party product coverage; historical patent litigation outcomes involving one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

In various embodiments, the analytics module 318 may form part of a system comprising a network; at least one database, accessible on the network, the database including at least one database of patents owned by the subject entity; a server, operatively connected to the network, wherein the server includes: a processor, a memory, software operable on the processor to: receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on selected external factors relating to one or more third parties external to the subject entity. The internal and external factors may be one or more of those described above.

In an embodiment, display module 320 is configured to display user interfaces and information retrieved from one or more databases 302-312. For example, display module 320 may generate user interface 200 of FIG. 2. If a user is accessing patent portfolio and annuity management system 102 (FIG. 1) remotely (e.g., through a web browser), display module 320 may be configured to transmit data representing a user interface through a network to a user terminal 104 (FIG. 1). In various embodiments, display module 320 is configured to generate one or more charts of data stored in databases 302-312. For example, display module 320 may generate a pie chart of the top 10 inventors within a portfolio.

In various embodiments, filtering module 322 is configured to filter a set of matters according to a user preference. For example, a user may activate a check box that indicates only pending matters should be shown in a user interface. Filtering module 322 may formulate a query consistent with retrieving only pending matters. The results of the query may then be shared with display module 320, where the user interface may be updated.

In various embodiments, mapping module 324 is configured to receive or generate mappings from a product to a matter. This may be accomplished, for example, by receiving a selection of a product and a matter by the user. The product may be one or more of the products stored in product database 310. In various embodiments, mapping module 324 is configured to present an automated mapping of the product to a matter for the user to confirm or deny. For example, mapping module 324 may present a claim chart for an issued patent to a user. As discussed, the claim chart includes one or more scope concepts to which one or more the claims of the patent are limited. In conjunction with input module 314, mapping module 324 may receive selections of the user of one or more of the scope concepts that are potentially affiliated with the product. Thus, a comparison may be made between what the claims are limited to and what the product may be. For example, if a product, as defined by the selection of one or more scope concepts by the user, is affiliated toward scope concepts A, B, and C, and claim 1 of the patent has been mapped to scope concepts A, B, and C, the mapping module 324 may present an option to map the product to the patent. In various embodiments, the user may override this recommendation and select to not map the product to the patent. Similarly, a user may indicate that a product is affiliated with the patent even if the scope concepts do not match.

In various embodiments, the mapping module 324 is configured to receive or generate mappings from prior art to a matter. This may be accomplished, for example, by receiving a selection of prior art and a matter by the user. The selection of the prior art may be based on cited references or through other external prior art search. The selections may be conducted manually, by a user, or automatically, by analytics module 318, for example. In various embodiments, mapping module 324 is configured to present an automated mapping of the prior art to a matter for the user to review. In various embodiments, the mapping module 324 can be configured to carry out any of the keyword analysis described above in relation to analytics module 318, or to map products stored in product database 310.

In an embodiment, once the product or prior art mapping information has been determined, it is stored within analytics database 312 or matter database 306. In various embodiments, queries are formulated and run on a periodic basis (e.g., nightly), and entries in analytics database 312 or matter database 306 may be updated to reflect any changes.

In various embodiments, payment module 326 is configured to receive selections, or tags, by the user of which annuities to pay for matters in a portfolio or registry listing. Additionally, payment module 326 may receive user preferences related to the payment of the annuities. Payment module 326 may receive payment information from a user and forward the payment information to the appropriate agency/office. In some embodiments, the annuity management system 102 (FIG. 1) organizes payment of the annuity on behalf of the user, and the user pays patent portfolio and annuity management system 102.

Any of the databases 302-312 or modules 314-326 described above may be used or configured partially or entirely as appropriate to perform one or more of the patent management functions described herein, or as set forth below in the following method steps.

Some embodiments of the present inventive subject matter include methods for assigning internal and external ratings to a subject patent owned by a subject entity.

One such embodiment is illustrated in FIG. 4. In this example embodiment, a method 400 for assigning internal and external ratings to a subject patent owned by a subject entity comprises: at block 402, assigning an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and, at block 404, assigning an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.

In an example embodiment, the internal factors may include one or more elements selected from the group comprising: perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

In an example embodiment, the method 400 may further comprise, at block 406, using one or more of the internal factors to identify a best subject patent and assigning a top internal rating to the best subject patent. The top internal rating assigned to the best subject patent may be in the range 8-10. The method 400 may further comprise, at block 408, assigning a relatively lower internal rating to at least one other patent owned by the subject entity. The relatively lower internal rating assigned to the at least one other patent may be in the range 0-7.

In an example embodiment, the external factors may include one or more elements selected from the group comprising: perceived deterrent value of the subject patent against the one or more third parties; third party product coverage; historical patent litigation outcomes involving the one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.

In an example embodiment, method 400 may include, at block 410, assigning the external rating to the subject patent by identifying one or more selected key patents owned by the one or more third parties, and assigning the external rating to the subject patent based on its relevance to, or relative worth when compared against, the one or more selected key patents of the one or more third parties. The external rating assigned to the subject patent may be in the range 0-10.

In example embodiments, one or more parts of the method 400 set out above are computer-implemented. The method 400 may therefore include the use of one or more processors to receive user input identifying the subject patent and assigning the internal and external ratings to the subject patent automatically on the basis of any one or more of the factors listed above.

In some example embodiments, the user provides additional input in analyzing the internal and external factors and weighting them to derive an overall internal and external rating for the subject patent.

Modules, Components and Logic

Certain embodiments are described herein as including logic or a number of components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied (1) on a non-transitory machine-readable medium or (2) in a transmission signal) or hardware-implemented modules. A hardware-implemented module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system) or one or more processors may be configured by software (e.g., an application or application portion) as a hardware-implemented module that operates to perform certain operations as described herein.

In various embodiments, a hardware-implemented module may be implemented mechanically or electronically. For example, a hardware-implemented module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware-implemented module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

Accordingly, the term “hardware-implemented module” should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily or transitorily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein. Considering embodiments in which hardware-implemented modules are temporarily configured (e.g., programmed), each of the hardware-implemented modules need not be configured or instantiated at any one instance in time. For example, where the hardware-implemented modules comprise a general-purpose processor configured using software, the general-purpose processor may be configured as respective different hardware-implemented modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware-implemented module at one instance of time and to constitute a different hardware-implemented module at a different instance of time.

Hardware-implemented modules can provide information to, and receive information from, other hardware-implemented modules. Accordingly, the described hardware-implemented modules may be regarded as being communicatively coupled. Where multiple of such hardware-implemented modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware-implemented modules. In embodiments in which multiple hardware-implemented modules are configured or instantiated at different times, communications between such hardware-implemented modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware-implemented modules have access. For example, one hardware-implemented module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware-implemented module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware-implemented modules may also initiate communications with input or output devices and can operate on a resource (e.g., a collection of information).

The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.

Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment, or as a server farm), while in other embodiments the processors may be distributed across a number of locations.

The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

Electronic Apparatus and System

Example embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, or software, or in combinations of them. Example embodiments may be implemented using a computer program product, e.g., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

In example embodiments, operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations can also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry (e.g., a FPGA or an ASIC).

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments deploying a programmable computing system, it will be appreciated that both hardware and software architectures usually require consideration. Specifically, it will be appreciated that the choice of whether to implement certain functionality in permanently configured hardware (e.g., an ASIC), in temporarily configured hardware (e.g., a combination of software and a programmable processor), or a combination of permanently and temporarily configured hardware may be a design choice. Below are set out hardware (e.g., machine) and software architectures that may be deployed in various example embodiments.

Example Machine Architecture and Machine-Readable Medium

FIG. 5 is a block diagram of a machine in the example form of a computer system 500 within which instructions for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), a tablet PC, a set-top box (STB), a PDA, a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system 500 includes a processor 502 (e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory 504 and a static memory 506, which communicate with each other via a bus 508. The computer system 500 may further include a video display unit 510 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computer system 500 also includes an alphanumeric input device 512 (e.g., a keyboard), a user interface (UI) navigation or cursor control device 514 (e.g., a mouse), a disk drive unit 516, a signal generation device 518 (e.g., a speaker) and a network interface device 520.

Machine-Readable Medium

The disk drive unit 516 includes a machine-readable medium 522 on which is stored one or more set of data structures and instructions 524 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions 524 may also reside, completely or at least partially, within the main memory 504 and/or within the processor 502 during execution thereof by the computer system 500, with the main memory 504 and the processor 502 also constituting machine-readable media.

While the machine-readable medium 522 is shown in an example embodiment to be a single medium, the term “machine-readable medium” may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more data structures or instructions 524. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing, encoding, or carrying instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the embodiments of the present invention, or that is capable of storing, encoding or carrying data structures utilized by or associated with such instructions. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories and optical and magnetic media. Specific examples of machine-readable media include non-volatile memory, including, by way of example, semiconductor memory devices (e.g., Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices); magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

Transmission Medium

The instructions 524 may further be transmitted or received over a communications network 526 using a transmission medium. The instructions 524 may be transmitted using the network interface device 520 and any one of a number of well-known transfer protocols (e.g., Hypertext Transfer Protocol “HTTP”). Examples of communication networks include a LAN, a WAN, the Internet, mobile telephone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., WiFi and WiMax networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such software.

Non-Limiting Embodiments

Although an embodiment has been described with reference to specific example embodiments, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the disclosure. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof show, by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. This Detailed Description, therefore, is not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred to herein, individually and/or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Note on the Abstract

The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b), requiring an abstract that will allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment. 

What is claimed is:
 1. A computer-implemented method of assigning internal and external ratings to a subject patent owned by a subject entity, the method including using one or more processors to: receive user input identifying the subject patent; assign an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and assign an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.
 2. The method of claim 1, wherein the internal factors include one or more elements selected from a group comprising: perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.
 3. The method of claim 2, further comprising using one or more of the internal factors to identify a best subject patent and assigning a top internal rating to the best subject patent.
 4. The method of claim 3, wherein the top internal rating assigned to the best subject patent is in a range 8-10.
 5. The method of claim 3, further comprising assigning a relatively lower internal rating to at least one other patent owned by the subject entity.
 6. The method of claim 5, wherein the relatively lower internal rating assigned to the at least one other patent is in a range 0-7.
 7. The method of claim 1, wherein the external factors include one or more elements selected from a group comprising: perceived deterrent value of the subject patent against the one or more third parties; third party product coverage; historical patent litigation outcomes involving the one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.
 8. The method of claim 1, wherein assigning the external rating to the subject patent includes identifying one or more selected key patents owned by the one or more third parties, and assigning the external rating to the subject patent based on its relevance to, or relative worth when compared against, the one or more selected key patents of the one or more third parties.
 9. The method of claim 7 or 8, wherein the external rating assigned to the subject patent is in a range 0-10.
 10. The method of claim 1, wherein the method includes assigning the internal and external ratings to the subject patent automatically and presenting the ratings to the user in a user interface or web page.
 11. A system for rating a subject patent owned by a subject entity, the system comprising one or more processors to: receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.
 12. The system of claim 11, wherein the one or more processors are to use one or more internal factors selected from the group comprising: perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.
 13. The system of claim 11, wherein the one or more processors are to use one or more external factors selected from a group comprising: perceived deterrent value of the subject patent against the one or more third parties; third party product coverage; historical patent litigation outcomes involving the one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.
 14. A system for rating a subject patent owned by a subject entity, the system comprising: a network; at least one database, accessible on the network, the database including at least one database of patents owned by the subject entity; a server, operatively connected to the network, wherein the server includes: a processor, a memory, software operable on the processor to: receive user input identifying a subject patent case to be rated; automatically assign an internal rating to the subject patent based on one or more selected internal factors relating to the subject entity; and automatically assign an external rating to the subject patent based on one or more selected external factors relating to one or more third parties external to the subject entity.
 15. The system of claim 14, wherein the internal factors include one or more elements selected from a group comprising: perceived relevance of the subject patent to subject company technologies; perceived relative value of the subject patent compared to other subject company patents; subject company product coverage; subject company patent marking lists; historical subject patent litigation outcomes involving the subject entity; historical subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications.
 16. The system of claim 14, wherein the external factors include one or more elements selected from a group comprising: perceived deterrent value of the subject patent against the one or more third parties; third party product coverage; historical patent litigation outcomes involving the one or more third parties; subject patent validity challenge outcomes; and how many times the subject patent has been cited against other patents or patent applications. 